feat(orchestration): introduce orchestration system#408
Open
Fodesu wants to merge 61 commits into
Open
Conversation
Adds a Writer wrapper around the blackboard Store that codifies the PLAN.md Stage 2.2 ownership table: - Workers may only write under their own task scope and may not touch the verifier namespace. - Verifiers may only write the verifier namespace at run or task scope. - Orchestrator may write any scope. result.* writes always require CompareAndSwap. The Writer rejects bare Put on result.* with ErrCASRequired, and CompareAndSwap fences stale writers whose ClaimEpoch is older than the value already in the store (ErrStaleWriter), even when the revision matches. WriterIdentity bundles the writer_type / writer_id / task_id / attempt_id / claim_epoch the authorisation layer needs and is validated up front by NewWriter. A small Reader wrapper mirrors Writer for symmetry in dependency injection. Backends are still free to expose Store directly. Tests cover orchestrator/worker/verifier authorisation, foreign-task rejection, cross-namespace rejection, CAS-required Put rejection, stale claim_epoch fencing, revision conflict propagation, and the Reader round-trip.
Implements Store on top of a NATS JetStream KV bucket. The bucket name
defaults to MEMOH_ORCH_BLACKBOARD; the entire orchestration runtime view
lives in one bucket with run/task scopes namespaced inside via the
canonical bb.{scope}.{owner}.{namespace}[.path...] key form.
Revisions are passed straight through to JetStream so CompareAndSwap
maps to the bucket's optimistic-concurrency primitives without any
client-side emulation. Insert (expected==0) maps to KV Create; non-zero
expected maps to KV Update; both surface CAS conflicts as
ErrRevisionConflict regardless of whether the underlying error is
ErrKeyExists or APIError(StreamWrongLastSequence).
List opens a filtered watcher over prefix and prefix.>, drains until
JetStream signals initial-sync complete, and rebuilds canonical Keys
from the raw KV key strings. Delete leaves a NATS tombstone so the
entry is hidden from List and surfaced as ErrNotFound on Get.
A small factory selects in-memory or JetStream based on an empty URL,
mirroring the orchestrationbus factory shape so callers can stay on the
in-memory backend in tests and stand-alone CLI tools.
Round-trip, CAS, list, and delete tests run against an actual NATS
server when TEST_NATS_URL is set (mirrors the TEST_POSTGRES_DSN gating
already used by other Postgres-backed tests). Buckets are deleted on
test cleanup so devenv state stays clean.
…spatch
Wires the kernel into the Stage 2 blackboard runtime view.
- Service.SetBlackboardStore registers an optional Store. When unset,
every blackboard call site short-circuits and behavior is identical
to the pre-Stage-2 kernel; tests and callers that have not opted in
see no change.
- On task completion the kernel publishes
bb.task.{task_id}.result.summary as the orchestrator-class writer.
CAS keeps repeated commits idempotent without threading the worker's
claim epoch through every kernel call site. Postgres remains the
authoritative copy and blackboard publish failures are logged only.
- On task dispatch the kernel snapshots the live revisions of run-scope
context plus the dependencies' result.* keys into
orchestration_input_manifests.captured_blackboard_revisions, so
verifier replay can later reconstruct the same view.
cmd/agent provides the JetStream- or in-memory-backed Store via the
existing [nats] config block and wires it into the kernel through an
fx.Invoke. Single-process deployments and tests stay on the in-memory
backend through the empty-URL factory fall back.
Integration coverage: TestIntegrationBlackboardCaptureAndPublish drives
a full producer/consumer DAG through the kernel and asserts both the
publish path (root + producer result entries appear in the store) and
the capture path (consumer's manifest carries the producer's blackboard
revision).
Adds the recovery primitive and admin entry point that close out the
Stage 2 contract. After a JetStream KV bucket loss, an operator can
invoke this against a quiesced run to repopulate the runtime view from
the authoritative Postgres copy without manual intervention or replay.
Service.RebuildBlackboard validates run access via the standard caller
identity, then walks two Postgres sources:
- orchestration_runs.goal/input/control_policy/source_metadata writes
bb.run.{run_id}.context.goal as the run-scope snapshot a worker would
see at dispatch.
- orchestration_task_results joined with the producing attempt writes
bb.task.{task_id}.result.summary entries, matching the payload shape
the live publish path produces during CompleteAttempt.
Each write goes through the orchestrator-class Writer with CAS against
the current revision, so concurrent live writes still reject the
rebuild attempt instead of silently overwriting fresher state. Per-key
errors are counted in the result and logged but do not abort the
rebuild; callers can compare write_errors vs the totals to decide
whether to retry. When no Store has been wired the call is a no-op
that returns blackboard_configured=false so HTTP callers can detect the
deployment shape from the response rather than guessing.
The new HTTP route POST /orchestration/runs/{run_id}/blackboard/rebuild
is declared on the existing orchestration handler interface, gets a
fake binding in the handler tests, and ships through swag + the
TypeScript SDK regenerated by openapi-ts.
Integration coverage:
- TestIntegrationRebuildBlackboardRecoversAfterStoreWipe wipes the
in-memory store mid-run, calls RebuildBlackboard, and asserts both
run-context and task-result entries reappear with the orchestrator
writer identity.
- TestIntegrationRebuildBlackboardWithoutStoreReturnsConfiguredFalse
pins the no-store contract so deploys without NATS keep returning
the same negative signal.
…ointer Walk PLAN.md through what actually shipped: namespace layout, writer ownership, frozen InputManifest capture at dispatch, post-commit result.summary publish, and the Postgres-backed RebuildBlackboard with its admin endpoint and SDK binding. Call out the two open follow-ups (JetStream wipe-and-recover blackbox harness, artifact/verifier rebuild) without conflating them with the contract that just landed, and roll the Next Step pointer forward to Stage 3 env sessions since Stage 2 CAS now unblocks moving control transitions onto the bus.
Lay down the five tables that anchor the env session runtime per RFC.md: env_resources describe leasable templates (container image, browser context, etc.) tenant-scoped with a capacity, env_sessions track concrete leased instances and carry lease_epoch fencing analogous to attempt claim_epoch, env_lease_reservations carry the admission ticket through reserve/commit/abort so the scheduler can queue when capacity is saturated, env_bindings map a session to the task/attempt currently using it (with a partial unique index that lets a session outlive a single attempt only when held for HITL resume), and env_snapshots record point-in-time captures keyed by session for verifier replay and drift detection. Indexes back the lookups the runtime layer will need next: per-resource status scans for capacity decisions, lease expiry scans for the reclaim loop, attempt/run lookups for binding inspection, and the prioritised pending queue on reservations. The active-binding partial unique index encodes the invariant that an active or held session has at most one binding at a time. The incremental migration in 0082_add_orchestration_env_runtime is mirrored in 0001_init.up.sql per the project rule. Both up and down were exercised end-to-end against a fresh postgres database (memoh-server migrate up / down / up). sqlc generated Go bindings for each new table and a focused query set covering create / get / list / update lifecycle plus the lease expiry sweep and the pending reservation queue. No runtime code uses these tables yet. S3-B introduces the internal/env package and the EnvManager interface that maps these rows into reserve/commit/abort/bind/snapshot/release semantics.
Add the package that wraps the Stage 3 env_* tables behind a thin
runtime surface. The Manager owns the durable state machine —
register a Resource, AcquireSession (which writes a reservation row,
inserts a session row in 'reserved' state, calls a Backend.Allocate
out of band, then commits both), CreateBinding / HoldBinding /
ResumeBinding / ReleaseBinding for task→session attachment, and
CaptureSnapshot / RenewSessionLease / ReleaseSession /
ReclaimExpiredSessions for the rest of the lifecycle. Every state
transition validates a (lease_token, lease_epoch) tuple against the
session row, mirroring the orchestration kernel's claim_epoch model
so a stale holder cannot mutate state behind the kernel's back.
Backend is the small driver-facing contract (Kind / Allocate /
Snapshot / Release). Backends register against a BackendRegistry by
kind; the manager refuses to allocate against an unregistered kind
(ErrBackendUnavailable) so a misconfigured deployment fails loud at
the dispatch boundary rather than silently fall back to an in-memory
shim. NoopBackend ships in this commit for tests and for
single-process deployments that have no real env runtime wired in
yet — Stage 3-C and 3-D add the container and browser drivers.
Notable design choices documented in code:
- Acquire is two-phase against Postgres (reservation+session locked
in step 1, backend.Allocate runs out of band, commit lands in step
2) so a slow backend never holds row locks. A crash between steps
leaves rows the reclaim sweep can finalise.
- ResumeBinding bumps lease_epoch and rotates lease_token — the
fence that separates a held HITL attempt from its resumed
successor.
- Capacity is enforced with a SELECT count + CHECK at the
reservation step. Stage 4 admission queueing will swap that for
fairness scheduling; today the call returns ErrCapacityExceeded
and the caller decides whether to back off.
- Bindings have a partial unique index on (session_id) WHERE status
IN ('active','held'), so the schema enforces the invariant that an
active or held session owns at most one binding at a time.
Renamed the package directory to internal/orchestrationenv after
discovering env/ is .gitignored at the repo root for python venv
conventions; the import path matches the Stage 1 / Stage 2 lineage
(orchestrationbus, orchestrationblackboard).
Tests cover happy-path acquire/release, capacity exceeded, stale
lease rejection on release, hold/resume epoch+token rotation, snapshot
capture and listing, expired-session reclaim, and duplicate binding
rejection. Each test creates its own postgres database and runs the
full migration set so coverage exercises the real schema, including
the deferred root_task_id FK on orchestration_runs.
Wire the orchestrationenv.Backend implementation for KindContainer
resources. The backend depends on a small Runtime interface
(PullImage / CreateContainer / StartContainer / StopContainer /
DeleteContainer / CommitSnapshot) that mirrors the subset of
internal/container.Service used at the env-session layer. Keeping the
dependency narrow means this package stays unit-testable with a fake
runtime — the cmd/agent wiring (Stage 3-E) is what binds it to
containerd, docker, or apple drivers in production.
Behaviour highlights:
- Allocate respects an image_pull_policy (always / if_not_present /
never) before CreateContainer, then starts the workload. A start
failure triggers best-effort DeleteContainer so a half-created
container does not leak the capacity slot.
- Container IDs and storage keys are derived deterministically from
the env_session_id ("envs-<id>"/"envs-rw-<id>") so retries reattach
to the same container/storage and operators can correlate
containers back to env sessions at a glance via the
memoh.orchestration_env.* labels stamped onto every container.
- Snapshot maps onto SnapshotService.CommitSnapshot. Backends that
return container.ErrNotSupported (apple virtualization today) are
surfaced as runtime-ref-only results with an "unsupported": true
flag so the env manager can still record metadata for inspector
views; real failures propagate to the caller.
- Release stops the container, tolerates stop failures (the runtime
may have already exited), and surfaces delete failures so the
manager can decide whether to retry. Container.ErrNotFound at
either step is treated as success since the runtime intent
(gone) is satisfied.
ResourceConfig fields the backend honours today: image / image_ref,
storage_driver / snapshotter, cmd, env, workdir, user. Stage 3-E will
extend this as the kernel adds mount and network requirements.
Tests cover the happy path, missing-image rejection, start-failure
cleanup, snapshot success and unsupported and error paths, release
under stop failure / delete failure / not-found, and the never-pull
policy.
Wire the orchestrationenv.Backend implementation for KindBrowser resources. Allocate POSTs to the existing apps/browser (Bun/Elysia/Playwright) gateway's /session endpoint and persists the returned ws_endpoint, gateway_session_id, and session_token into the env session's runtime handle so the worker can drive Playwright remotely through the gateway. Release closes the gateway session via DELETE /session/:id?token=... The package depends on a small Gateway interface so unit tests stay independent of the real Bun service. NewHTTPGateway ships the production client and matches the on-the-wire schema documented in apps/browser/src/modules/session.ts; tests round-trip against a real httptest.Server to confirm the request/response shapes match. Snapshot is intentionally minimal in this stage. The browser gateway exposes no native snapshot endpoint — capturing browser state means playing Playwright scripts against the live session, which is a worker-side action rather than a backend primitive. The backend returns a stable bookkeeping ref (gateway_session_id, ws_endpoint, snapshot kind) marked unsupported=true so the env manager still records snapshot rows for inspector views; Stage 3-I replaces this with real cookie / storage / screenshot capture once drift detection is designed. Resource config knobs honoured today: core (chromium / firefox), ttl_ms, context_config (passed through verbatim). bot_id sent to the gateway is derived from env_session_id with a configurable "envs-" prefix so env-managed sessions stay distinguishable from real bot sessions in gateway logs and per-bot quotas. Tests cover happy-path allocate, default-core fallback, gateway error propagation, snapshot bookkeeping, release with present and empty handles, release error propagation, plus HTTP gateway round-trip and HTTP error status handling.
…anifests
Add two JSONB columns that Stage 3-E uses to drive env session
reservation. orchestration_tasks.env_preconditions captures what the
planner declared for the task ("does this task need a container or
browser session, and which kind?"). orchestration_input_manifests.
captured_env_preconditions captures what the kernel actually resolved
at dispatch time, so a verifier replay can rebuild the same lease
context the worker saw.
Both columns default to '{"required": false}' so the column stays
NOT NULL without breaking existing rows or creating a NULL special
case in readers. defaultEnvPreconditionsJSON feeds that sentinel into
every CreateOrchestrationTask and CreateOrchestrationInputManifest
call site (root task, planner-emitted child tasks, dispatch manifest,
plus all integration test seeds), so behaviour is unchanged in this
commit — the columns exist, sqlc has binding code for them, and
nothing reads them yet.
Subsequent S3-E commits add the planner contract, the dispatch hook
that calls envManager.AcquireSession / CreateBinding, the manifest
hash inclusion, the cmd/agent FX wiring, and the admin HTTP CRUD
surface for env resources.
Promote env_preconditions from a deferred Stage 3 idea to a mandatory
field on the planner contract. Every child task the start-run planner
or replanner emits must now declare whether it depends on a leasable
runtime environment. Required=false marks pure-LLM steps and remains
the common case; Required=true demands kind ∈ {container, browser}
and a non-empty resource_name so the kernel can later resolve the
operator-managed env_resource and reserve a session before dispatch.
Domain side:
- internal/orchestration/types.go gains an EnvPreconditions struct
(with Required/Kind/ResourceName/Mode/EffectClass/Metadata) plus
EnvPreconditionsKind* and EnvPreconditionsEffect* constants that
mirror the orchestration_env_resources / action ledger CHECK
vocabulary. PlannedTaskSpec and Task now carry it as a first-class
field, and toTask projects the env_preconditions JSONB column back
into the struct.
- runtime.go threads the planner-supplied value through the
materialise path (replacing the default sentinel from S3-E.1),
through the replacement_plan payload, and through
plannedChildTasksFromSpecs / plannedChildTasksFromReplacementPlan.
decodeEnvPreconditionsObject and normalizeEnvPreconditions
centralise the structural validation: required=false strips every
optional field, required=true enforces kind/resource_name and
rejects unknown keys.
Planner contract side:
- internal/orchestrationexec/start_run_planner.go now requires
env_preconditions on every emitted child task, with
requiredPlannerEnvPreconditions / decodePlannerEnvPreconditions
performing the same validation as the kernel-side decoder.
- The system prompts for both the start-run planner and the
replanner explicitly describe the env_preconditions schema and
remind the model to mark required=true only for tasks that touch
an external runtime.
Tests:
- New llm_runtime_test.go cases cover the happy path, the
env-bound container shape, and three rejection paths (missing
field, invalid kind, required=true without resource_name).
- TestDecodeStartRunPlannerPayloadValidatesChildTasks and
TestDecodeReplanPlannerPayloadUsesStrictChildTaskSchema now pass
env_preconditions through so they exercise the new decoder.
- The blackbox harness LLM stub emits env_preconditions=false on
every fake child task it returns, keeping the runtime smoke
tests green under the tighter contract.
Replan payloads that pre-date Stage 3-E may still omit the field;
plannedChildTaskEnvPreconditions falls back to required=false in that
case so historic replans stay replayable without a data migration.
…mpletion The kernel now drives the env session runtime end to end. When a planner emits env_preconditions.required=true the dispatcher resolves the resource_name on the EnvManager, acquires a session, creates a primary binding for the attempt, and persists the captured envelope into captured_env_preconditions on the dispatch manifest. CompleteAttempt releases both the binding and the underlying session once the attempt reaches a terminal state, and dispatch aborts run a best-effort compensation against the env manager so a failed dispatch does not strand a session. The reclaim sweep remains the authoritative backstop for any state that loses the race. EnvManager is exposed as a primitive-typed interface in internal/orchestration so the kernel does not depend on the orchestrationenv package, which lets cmd/agent (S3-E.4) plug in the real Manager without a circular import and lets unit tests substitute a fake. The dispatch path threads an internal envCapture struct through the manifest so a verifier replay can fence stale callers without re-resolving the planner-supplied resource. Tasks with required=false take a fast path that never touches the env manager, keeping pure-LLM dispatches byte-identical with the column default. Integration coverage exercises both branches against a real database with a fake EnvManager, asserting the call sequence, the lease fencing tuple, the manifest capture, and the post-completion release.
…sweeps The orchestration kernel now sees a real environment session manager when it runs as part of the unified server. cmd/agent constructs the env backend registry from the same container service and browser gateway the rest of the server uses, instantiates orchestrationenv.Manager from the shared Postgres pool, and adapts it through KernelAdapter so the kernel keeps consuming the primitive orchestration.EnvManager interface declared in S3-E.2. A periodic reclaim loop sweeps expired sessions on a 30-second cadence so that lease TTLs alone do not strand env_resources after a worker crashes between dispatch and release. The default lease TTL stays at the kernel's 30-minute floor and operators can override it via MEMOH_ORCHESTRATION_ENV_LEASE_TTL_SECONDS without rebuilding. KernelAdapter lives in internal/orchestrationenv so the kernel never has to import the env package. The container backend opts in via a runtime type assertion against the existing ctr.Service value: deployments whose container service does not satisfy the wider env runtime surface skip container env resources rather than refusing to boot, and deployments without a browser gateway skip the browser backend the same way. Missing backends surface as ErrBackendUnavailable on dispatch, which keeps the misconfiguration visible at the call site instead of papering over it.
Expose tenant-scoped env resource CRUD so operators can manage runtime templates before planner dispatch depends on them.
Classify orchestration actions and attach env session plus snapshot references so env-backed attempts are auditable from dispatch through release.
Hold env bindings across resume_held_env checkpoints and reattach them to the next attempt with a rotated lease so HITL resumes keep the same runtime safely.
Flatten planner and completion tool calls so LLM outputs fail in controlled validation paths, and keep browser automation modeled as an env with context/exclusive modes.
Add the orchestration sidebar, env resource pages, image management views, and updated run details so the frontend matches the expanded orchestration APIs.
Reuse the chat tool-call presentation in the run inspector so orchestration actions stream into a familiar, compact Act tab.
Use orchestration intent terminology across storage, runtime, API contracts, and generated clients so control intents are no longer modeled as planner-only lifecycle state.
Replace the monolithic orchestration run view with Vue Flow based DAG and timeline views, including lane-local time compression and clearer status metadata.
Use theme-aware neutrals for orchestration chrome while keeping selected tasks readable with a subtle primary accent.
Move orchestration control-plane loops into a dedicated orchestrator process so the API server no longer owns NATS-backed runtime coordination.
Consolidate the branch's orchestration schema changes into a smaller migration sequence so review and fresh database setup stay manageable.
Add task-scoped cancellation and retry APIs, including durable requeue support for failed start-run planning so planner failures can recover through the same task control path.
Place retry actions directly on failed DAG and flow task cards, and keep the environment inspector focused on the environment name and type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#391
本 PR 为 Memoh 引入面向长复杂任务的编排执行层,使任务具备可观察、可恢复、局部重试、验证与审计能力,从而提升用户对真实任务托付的信任。
Memoh 已经具备成熟的 chat-first bot runtime:对话、工具调用、记忆注入以及 bot workspace 内的命令执行都可以在现有运行路径中完成。这条路径适合对话式协作和短周期任务。
长复杂任务需要的不只是一次模型回复,而是一个可被管理的运行对象。这个 PR 引入 orchestration runtime,将这类任务从聊天上下文中独立出来:User Bot 仍然负责用户交互,run 自身拥有状态机、worker、verifier 和 Web UI。
本 PR 的 review 主线是 durable run/task execution layer。NATS、blackboard、EnvSession 等内容服务于这条主线,是围绕该执行层建立的初版 runtime contracts,不代表完整 autonomous task platform 已经完成。
Why:长复杂任务需要执行层
长复杂任务不是一次问答,而是一段会随中间结果改变路径的执行过程。它可能拆分出多个依赖步骤,某些步骤失败后需要局部修正,某些结果需要验证后才能进入下一步。对这类任务,聊天上下文适合作为交互入口,但不适合作为执行状态本身。
可观察性
真实任务不能只留下最终回复。用户和维护者需要了解任务被拆分成哪些步骤、当前停留在哪个阶段、哪些 attempt 已完成、哪些 checkpoint 等待处理,以及哪些 artifact 或 env 状态参与了结果生成。
Orchestration 将 run、task、attempt、event、checkpoint、verification 建模为可查询对象。Web UI 提供 DAG、Flow timeline 和 task inspector,使维护者能够回答:任务为什么停在这里、谁持有 lease、哪个 attempt 失败、失败后是否触发了 retry/replan,以及最终结果经过了哪次 verification。UI 在这里是 trust surface,不只是 debug tooling。
可恢复性
长任务需要应对进程重启、lease 过期、worker 离线、verification 中断、人工等待超时等情况。聊天上下文只能恢复叙述,持久化 run state 才能恢复调度和执行进度。
本 PR 使用 Postgres 记录权威状态,通过 worker/verifier lease fencing 阻止过期执行者继续写入,并通过 recovery loop 接管过期的 attempt、verification 和 checkpoint。任务中断后,系统可以判断应继续、重试、等待还是失败。
局部重试
复杂任务通常由多个相互依赖的步骤组成。某个步骤失败时,不应默认丢弃整个 run,也不应要求用户手动判断哪些结果仍可复用。
Orchestration 将任务拆分为 DAG,并将 retry、replan、supersede 表达为显式状态迁移。失败发生在 task/attempt 层,系统可以仅重做需要重做的分支,并保留已经有效的结果。
验证与审计
长任务交给 agent 执行之后,用户需要的不只是"模型声明完成"。结果是否可接受、失败来自工具还是任务本身、谁在何时执行了什么,都应进入 run 生命周期。
本 PR 引入 verifier runtime、verification records、committed run events、runtime facts 和 side-effect approval token,使 agent output 从"最终回答"变成可被 accept/reject 的 claim。验证结果、运行事实和事件记录也会成为后续 eval、资源治理和经验沉淀的输入。
Who:组件职责
cmd/agent):负责 HTTP API、鉴权以及 Web UI 读路径。不依赖 NATS 启动;mutating orchestration API 会检查 orchestrator 是否在线,控制平面不可用时返回503。cmd/orchestrator):控制平面 daemon,负责 planning、scheduling、recovery、verification recovery、event outbox、fact consumer 和 env reclaim。cmd/workerd) / Verifier (cmd/verifyd):task-scoped runtime,分别负责执行 attempt 和验证 completed attempt。What:这个 PR 引入的运行模型
本 PR 引入一套持久化 run 模型:
当前实现已覆盖主路径:创建 run、规划 DAG、调度 attempt、执行任务、验证结果、从 lease loss 恢复、等待用户输入,并在 UI 中查看完整过程。
When:适用场景
并非所有任务都需要进入 orchestration。更适合使用该路径的任务包括:
不适合进入 orchestration 的任务应继续留在 chat/runtime 路径中,例如一次性问答、短工具调用、无需验证的轻量操作、没有跨步骤状态的任务。Orchestration 会引入额外状态、daemon 和 UI 面,只有当任务价值足以覆盖这些成本时才值得使用。
典型生命周期如下:
Where:代码与部署边界
主要代码位置:
internal/orchestration:持久化状态机和 service APIinternal/orchestrationexec:planner、replanner、worker runtime、verifier runtimecmd/orchestrator:控制平面 daemoncmd/workerd:worker daemoncmd/verifyd:verifier daemoninternal/orchestrationbus:event/fact bus contract 和 JetStream 实现internal/orchestrationoutbox:committed event publisherinternal/orchestrationfacts:fact validator/consumerinternal/orchestrationblackboard:可重建的运行期视图internal/orchestrationenv:environment session runtimeinternal/handlers/orchestration.go:HTTP APIapps/web/src/pages/orchestration:DAG、Flow、inspector UICompose 部署新增
nats、orchestrator、workerd和verifyd。这些进程对应不同职责:API server 负责产品入口,orchestrator 负责控制平面,worker 和 verifier 负责任务执行与验证。拆分进程可以明确故障边界和扩缩容边界。How:核心设计原则
这个设计要解决的不是“怎么多跑几个 worker”,而是长任务在失败、重启和重新规划之后仍然能解释清楚。原则按四类约束组织:事实写在哪里、谁推进状态、失败后怎么恢复,以及哪些东西只用于观测或运行现场。
状态与历史
控制面与执行面
cmd/agent是用户/API/UI 边界,负责鉴权、参数校验、读路径和用户触发的写操作。控制面不可用时,mutating orchestration API 返回503,而不是静默降级。cmd/orchestrator消费。这样控制动作可以恢复、重试,也可以被审计。cmd/workerd执行 attempt,cmd/verifyd验证 completed attempt。它们不决定整个系统的事实来源,只提交受 lease/fence 约束的状态迁移。恢复与并发控制
运行时契约与观测
How it works:运行模型
这里按语义拆成多张图,而不是把所有关系塞进一张图:组件边界图说明谁负责什么,创建链路图说明用户请求如何落库,生命周期图说明 run/task 状态如何流转,runtime facts 图说明旁路观测如何校验。
组件边界
这张图只回答组件职责,不表达生命周期。
cmd/agent是用户和 Web 的 API 边界;cmd/orchestrator是长任务控制面;cmd/workerd和cmd/verifyd是任务运行时;Postgres 是唯一事实来源;NATS、blackboard 和 env session 是运行期契约,不是权威状态。flowchart LR entry["User-facing API\nUser Bot / Web UI / cmd/agent"] postgres["Postgres\nauthoritative run state"] control["Control plane\ncmd/orchestrator"] runtime["Task runtime\ncmd/workerd / cmd/verifyd"] contracts["Runtime contracts\nNATS / Blackboard / Env sessions"] entry -->|"create, query, cancel, retry"| postgres control -->|"plan, schedule, recover"| postgres runtime -->|"attempt and verification transitions"| postgres control -->|"assigns work through persisted state"| runtime runtime -->|"publishes facts, uses execution context"| contracts control -->|"publishes events, manages env leases"| contracts创建任务链路
用户仍然从 bot 对话入口发起任务。这个过程是请求时序,不是组件依赖图:User Bot 调用 orchestration tool,API server 负责鉴权和写入请求,最终在 Postgres 中创建 run、root task 和
start_runintent。sequenceDiagram actor User participant Bot as User Bot participant Tool as orchestration tool participant API as API server cmd/agent participant DB as Postgres User->>Bot: ask for a long-running task Bot->>Tool: create orchestration run Tool->>API: POST CreateRun API->>API: validate request and authorize caller API->>DB: insert run, root task, start_run intent DB-->>API: run handle API-->>Tool: run_id and snapshot_seq Tool-->>Bot: run createdRun 生命周期
生命周期应该画成状态机。Orchestrator 推进 run 和 task 的权威状态;worker/verifier 只是通过带 lease/fence 的状态迁移提交执行和验证结果。失败不一定终止整个 run,它可能进入局部 retry、replan 或等待用户输入。
stateDiagram-v2 [*] --> Created Created --> Planning: start_run intent Planning --> Running: DAG committed Planning --> Failed: planner exhausted Running --> Dispatching: ready task claimed Dispatching --> AttemptRunning: worker lease acquired AttemptRunning --> Verifying: attempt completed AttemptRunning --> Retrying: attempt failed and retryable AttemptRunning --> Failed: attempt failed and terminal Verifying --> TaskCompleted: result accepted Verifying --> Retrying: verifier requests retry Verifying --> Replanning: verifier requests replan Verifying --> WaitingHuman: checkpoint required Verifying --> Failed: verification failed and terminal Retrying --> Dispatching: new attempt Replanning --> Planning: replan intent WaitingHuman --> Running: checkpoint resolved WaitingHuman --> Failed: checkpoint timeout TaskCompleted --> Running: more ready tasks TaskCompleted --> Completed: all tasks completed Running --> Cancelling: cancel requested Dispatching --> Cancelling: cancel requested AttemptRunning --> Cancelling: cancel requested WaitingHuman --> Cancelling: cancel requested Cancelling --> Cancelled Completed --> [*] Failed --> [*] Cancelled --> [*]Runtime facts 与观测
runtime facts 是旁路观测,不是当前的主状态写入路径。Worker 和 verifier 把 attempt/verification fact 发到 NATS;fact consumer 读取这些 fact,并对照 Postgres 校验 orphan、stale claim 和 identity mismatch。当前 fact consumer 是 read-only validation,不是 API server 的下游,也不是权威 writer。
flowchart LR worker["Worker\ncmd/workerd"] --> facts["NATS JetStream\nruntime facts"] verifier["Verifier\ncmd/verifyd"] --> facts facts --> consumer["Fact consumer\nread-only validation today"] consumer -.-> postgres["Postgres\nauthoritative state"] outbox["Committed event outbox"] --> events["NATS JetStream\ncommitted events"] events --> webUi["Web UI stream\nDAG / Flow / inspector"] postgres --> blackboard["Blackboard\nrebuildable runtime view"]Postgres 是事实来源,保存 run、task、intent、attempt、verification 和 committed event。NATS 只负责 runtime fact 和 committed event delivery;API server 不依赖 NATS 启动,NATS 丢失也不应改变已提交事实。Blackboard 是可重建的 runtime view,Env session 是浏览器或容器现场的运行时绑定,二者都不作为权威状态。
这个 PR 包含什么
持久化执行内核
控制面与任务运行时
cmd/orchestrator,负责 planning、scheduling、recovery、verification recovery、event outbox、fact consumer 和 env reclaim。cmd/workerd与cmd/verifyd,分别执行 task attempt 和验证 completed attempt。API 与产品面
503。运行时契约
RFC scope
这不是完整 RFC。本 PR 落地的是 core runtime 和 UI 基础。Quota / admission、artifact reservation、experience learning 仍属于后续工作。
Test plan
本分支已运行:
go test ./cmd/agent ./cmd/orchestrator ./internal/orchestration ./internal/orchestrationexec ./internal/handlersdocker compose -f docker-compose.yml config --quietdocker compose -f devenv/docker-compose.yml config --quietgit diff --check